{"id":"0a3bff50-8aa3-4724-9fa4-028196c21c57","arxiv_id":"2607.17790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ReViV jointly reconstructs body, hand, gaze, camera trajectory, and depth from monocular egocentric video using one masked generative transformer, outperforming prior methods on body, hand, and gaze while running much faster.","lead":"This paper presents ReViV, a single neural network that takes one egocentric video from a wearable camera and reconstructs the wearer's body, hands, gaze, camera path, and scene depth in 4D. It is a step toward always-on 3D understanding for smart glasses, AR/VR, and embodied AI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Metric 4D claim rests on an unvalidated scale-alignment step; all reported depth metrics are scale-invariant and cannot support the 'metric' part of the central claim.","rationale":"The reader's conditional verdict correctly identifies the depth pseudo-label and metric alignment pipeline as the weakest load-bearing component. I agree: the '4D' and 'metric-aligned' language in the title and abstract depend on a scale-recovery step that is not evaluated on absolute scale, and all reported depth numbers are invariant to global scale because of the clip-wise alignment protocol. This is the most load-bearing concern because it targets the central claim itself rather than individual benchmark rankings. The body reconstruction results on ADT, which was excluded from training, are comparatively well controlled and provide genuine support for the viewer-side contribution. Similarly, the camera tracking and gaze results are informative even though the abstract overstates 'SOTA' for camera tracking given the ATE gap to VIPE in Table 4. I did not choose the hand-benchmark contamination issue as the primary concern because the paper's own contribution, a unified generative framework, does not stand or fall on exact hand numbers, and the hand comparisons use official baseline implementations with standard evaluation settings. The metric-scale issue, by contrast, undermines the title claim of reconstructing the viewer and view in 4D. The recommended remedy is not rejection: the model, code, and data are released, so a direct scale-error evaluation is feasible and could settle the concern. Therefore the conditional verdict remains appropriate.","tokens_in":20369,"tokens_out":3433,"duration_ms":33775,"concrete_test":"Run the released ReViV model on ADT clips and apply the Sec. D alignment (floor-fitting or VIPE anchor). Compare the resulting metric-aligned depth maps directly to ADT ground-truth depth without any clip-wise scale/translation post-alignment, reporting median scale error and absolute relative error. Additionally, for each clip, compute the scale factor from the floor-fitting heuristic and from the VIPE anchor separately and report their per-clip ratio. If median scale error exceeds roughly 15% or the two alignment strategies disagree by more than 20% on a substantial fraction of clips, the metric 4D reconstruction claim is not supported by the current evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is a 'metric 4D reconstruction' of viewer and view. To obtain metric scale, Sec. 4.1 and Supplement D introduce a post-hoc alignment that fits a floor plane using the lowest 50% of body feet z-values and the lowest 5% of scene points, or falls back to VIPE metric depth anchors. This step is never validated. Every depth metric in Table 4 and Sec. 5.5 is computed after a clip-wise scale and translation alignment to ground truth, which removes global scale error entirely. Thus the reported Abs Rel and delta_1.25 numbers cannot confirm that the Sec. D alignment produces correct metric geometry. ADT provides metric depth and body ground truth, so an absolute-scale evaluation is possible, but none is reported. The pseudo-labels from Video Depth Anything (Sec. 3.1) are also not checked against ADT before training. If VDA has biased scale, the floor-fitting heuristic fails on sequences without a visible floor, or the VIPE anchor disagrees with the floor anchor, the 'shared metric 4D coordinate system' in Fig. 2 will be inaccurate even though all tables remain unchanged. This is not an internal inconsistency; it is an unsubstantiated assumption in the load path of the title claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReViV proposes a unified masked generative transformer (MGET) that takes a single monocular egocentric RGB video and jointly infers body motion, hand motion, gaze, camera trajectory, and affine-invariant depth, with a post-hoc alignment step to obtain a metric 4D reconstruction. The system tokenizes each modality with a VQ-VAE, trains on a 7B-token multimodal corpus assembled from existing datasets plus Video Depth Anything pseudo-labels, and decodes with iterative masked sampling. The paper reports held-out ADT evaluations for body, camera, gaze, and depth, plus hand evaluations on HoloAssist, HOT3D, ARCTIC, and TACO, and claims state-of-the-art accuracy and efficiency for holistic egocentric reconstruction, with code and models open-sourced.","tokens_in":20642,"tokens_out":2763,"duration_ms":26243,"significance":"If the results hold, the paper would be a notable step toward a single feed-forward, hardware-free egocentric 4D reconstruction system: body, hand, gaze, camera, and depth at 0.7 s per 2-second clip, with strong generalization to the held-out ADT benchmark. The main quantitative strengths are the body and hand results, which use standard metrics and beat recent diffusion-based and optimization-based baselines by large margins, and the explicit exclusion of ADT from training. The paper also contains systematic ablations (masking strategy, model capacity, data scaling, ViT branch) and ships open-source code, which increases reproducibility. However, the 'metric 4D' and 'state-of-the-art camera tracking' parts of the central claim are not supported by the reported experiments, and the body comparison protocol is not fully matched.","major_comments":[{"comment":"The paper's title and abstract claim a 'metric 4D reconstruction', but the depth evaluation in Table 4 cannot support this. All depth metrics are computed after a clip-wise scale-and-translation alignment to ground truth (Sec. 5.5), which removes global scale and offset errors; the metric alignment step described in Sec. 4.1 and Supplement D (floor percentiles, or VIPE anchors) is never evaluated against the metric ground truth that ADT provides. Since ADT has metric depth and body annotations, the authors should report an absolute-scale error (e.g., metric Abs Rel before alignment, scale-error distribution, or a comparison of the aligned reconstruction against the metric GT) to substantiate the 'metric 4D' claim. As written, the tables remain identical whether the Sec. D alignment is correct or systematically biased.","section":"Sec. 4.1, Sec. 5.5, Supplement D"},{"comment":"The abstract claims 'state-of-the-art ... camera tracking', but Table 4 reports ATE of 0.015 for ReViV versus 0.005 for VIPE, so VIPE is better on the primary absolute-trajectory metric. The paper acknowledges this gap in Sec. 5.3 but the abstract and contributions overstate the result. The claim should either be revised to 'competitive' or accompanied by evidence that the feedforward setting is the intended point of comparison rather than raw accuracy; the current wording is contradicted by the paper's own table.","section":"Sec. 5.3, Table 4, Abstract"},{"comment":"The body-motion SOTA claim is weakened by an uneven comparison protocol. EgoAllo and UniEgoMotion are not retrained on the same data; the evaluation protocol states that this is impossible for the authors because their pretraining corpus lacks SMPL-X annotations. However, the reported comparison therefore mixes model architecture, training data, and input modalities (ReViV uses full-image semantics and multi-modal pretraining, while baselines use camera trajectories). To support the 'SOTA from monocular video' claim, the paper should either provide a same-data retrained baseline (e.g., using a smaller annotated subset), report results on an additional unseen dataset, or explicitly frame the comparison as 'out-of-the-box baselines' and quantify the effect of the input-camera information via an ablation that feeds estimated trajectories to ReViV.","section":"Sec. 5.1, Table 2"},{"comment":"The depth pseudo-labels from Video Depth Anything are used for large-scale geometric pretraining (Sec. 3.1) but their quality is never validated against the ADT ground truth before training. If VDA has biased scale or if the floor-fitting/VIPE anchor fails on sequences without a visible floor, the 'shared metric 4D coordinate system' in Fig. 2 will be inaccurate even though all reported tables are unchanged because of the clip-wise alignment in Sec. 5.5. The authors should add a pseudo-label-quality analysis (e.g., scale-error distribution of VDA on ADT) or explicitly state that the metric 4D claim is conditional on the alignment step's validity, which is currently unverified.","section":"Sec. 3.1, Sec. 5.5"}],"minor_comments":[{"comment":"The table heading reads 'T able 1' due to a spacing error; please fix the typo.","section":"Table 1"},{"comment":"For the HoloAssist hand comparison, Dyn-HaMR is evaluated on a random 3,000-clip subset while ReViV and HaMeR are evaluated on the full 27,910-clip validation set; the paper should report the standard error or repeated random splits so that the comparison is not confounded by subset selection.","section":"Sec. 5.2"},{"comment":"The depth comparison would benefit from also reporting scale-invariant metrics (e.g., sc-inv or delta thresholds at multiple tolerances) to separate geometric quality from the alignment choice; the current pair of Abs Rel and delta_1.25 is rather coarse.","section":"Sec. 5.5, Table 4"},{"comment":"The inference details state that classifier-free guidance uses w=2 and top-p=0.8, but no sensitivity analysis is provided for these decoding hyperparameters; adding a small ablation would strengthen reproducibility.","section":"Supplement C"},{"comment":"The objective in Eq. (3) excludes the continuous ViT tokens from the prediction loss, but the main text says the model predicts 'all' masked tokens; please clarify the role of the xViT tokens in the masked prediction formulation.","section":"Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The paper is from the same group as EgoM2P, and several components are re-used and retrained from that prior work; this is not circularity, but the novelty relative to EgoM2P should be made crisper in the revision. The central architecture claim is credible for body and hand reconstruction, but the metric 4D and camera-tracking claims are currently overstated relative to the evidence. If the authors add the requested absolute-scale evaluation and soften the SOTA camera claim, the paper could become a strong contribution. The open-source release is a plus, but the statement 'code and models fully open-sourced' should be verified by the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ReViV is worth your time: it is the first single feed-forward model that jointly predicts body, hand, gaze, camera trajectory and depth from monocular egocentric video at 0.7 s per clip. The body and hand results on held-out ADT and four hand benchmarks are genuinely strong, and the ablations are more thorough than usual for this area. The masked generative transformer over unified body/hand tokenizers is a clean extension of EgoM2P, and the paper is honest about most of its design choices.\n\nThe soft spot is the 'metric 4D reconstruction' claim. Every depth metric in Table 4 is computed after a clip-wise scale and translation alignment to ground truth, so those numbers cannot confirm that the floor-fitting or VIPE-anchor alignment in Sec. 4.1/D actually produces correct metric geometry. ADT has metric depth and body ground truth; an absolute-scale evaluation is possible and should have been reported. As written, the metric part of the title and abstract is unsupported. That is a load-bearing overstatement, though it doesn't invalidate the body/hand/gaze results, which are independent of metric scale.\n\nTwo smaller issues. The SOTA claims are a bit broad: VIPE beats ReViV on ATE and EgoMono4D beats it on depth, so the abstract's 'state-of-the-art ... camera tracking' is overstated. And on body motion, baselines were not retrained on the same data and no error bars or significance tests are reported, so the size of the win is less certain than the table suggests, even though the margins are large enough that I expect they hold.\n\nOverall, this deserves a serious referee. I'd send it to review and ask for a revision that either validates the metric alignment on ADT's absolute scale or explicitly reframes the contribution as scale-ambiguous reconstruction, plus tempered camera and depth claims. The core model is a real advance for egocentric 4D perception.","headline":"Strong unified body/hand/gaze model, but the metric 4D claim is unsupported because all depth metrics are scale-invariant and the alignment step is never validated.","tokens_in":21224,"tokens_out":2822,"would_cite":true,"duration_ms":24341,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One first-person video yields the wearer's motion and a metric 4D scene in a single pass.","keywords":["egocentric vision","4D reconstruction","masked generative modeling","human motion estimation","hand pose estimation","gaze estimation","camera trajectory estimation","depth estimation"],"falsifier":"Take ReViV's predicted depth on the Aria Digital Twin sequences that have ground-truth metric depth, skip the usual clip-wise scale-and-shift alignment, and compare absolute scale; if the scale error is large precisely on sequences where the floor is visible, the floor-fitting step, not the kinematic predictions, is what carries the metric claim.","tokens_in":20152,"feed_emoji":"🕶️","tokens_out":6726,"duration_ms":55969,"temperature":0.7,"pith_summary":"The paper tries to show that a single monocular video from a head-mounted camera contains enough information to reconstruct, at the same time, the wearer's full-body motion, hand poses, gaze direction, the camera's trajectory, and the scene depth. It argues that these signals should not be predicted separately, because they constrain one another: where the wearer looks and reaches is tied to the scene geometry, and the camera motion is tied to the body. ReViV casts the whole reconstruction as one masked generative modeling problem over a shared discrete token space, and reports that a single feed-forward transformer, trained on a seven-billion-token multimodal corpus, reconstructs all modalities at 0.7 seconds per two-second clip. If the claim holds, wearable devices could build a metric 4D model of the user-and-world interaction directly from raw video, without SLAM or external hand trackers.","feed_headline":"One first-person video yields body, hands, gaze, and 4D scene","feed_subtitle":"A masked generative transformer predicts all five modalities in one 0.7-second feed-forward pass.","key_machinery":"The Masked Generative Egocentric Transformer (MGET), a T5-style encoder-decoder trained on a sequence of discrete tokens produced by modality-specific vector-quantized autoencoders (VQ-VAEs). Each continuous signal, be it body joints, hand joints, gaze, camera trajectory, RGB, or depth, is quantized into a codebook vocabulary; during training, random masks over the unified token sequence force the model to predict any subset of modalities from any other subset, thereby approximating the joint distribution $p(X,Y)$. At inference, only the RGB tokens and a continuous vision-transformer branch are visible, and the missing tokens are decoded iteratively with classifier-free guidance. The continuous branch preserves high-frequency visual detail that the discrete quantization would otherwise discard.","core_discovery":"The central claim is that body, hand, gaze, camera trajectory, and depth can be treated as one joint distribution over discrete tokens, and that a transformer trained to predict randomly masked tokens from this distribution learns the cross-modal couplings that make each individual prediction more accurate and temporally consistent. On the unseen Aria Digital Twin benchmark, ReViV reports lower aligned pose error and higher motion similarity for body reconstruction than diffusion baselines that are given ground-truth or estimated camera trajectories, and it reports lower hand pose errors than optimization-based hand trackers on HoloAssist, HOT3D, ARCTIC, and TACO. The camera trajectory and gaze predictions also improve on the egocentric pretraining baseline, while depth remains competitive with a specialized monocular depth model but does not surpass it. The metric 4D alignment is obtained by a lightweight post-hoc scale step: fitting a floor plane from the predicted body and scene, or anchoring to an off-the-shelf metric depth estimate when no floor is visible.","pith_inferences":["If the joint-distribution objective really is the source of the gains, the same tokenization could be pretrained on even larger unlabeled egocentric video, treating the depth pseudo-labels as one noisy modality among several rather than as ground truth.","A natural stress test would be measuring metric scale error on long recordings with known floor heights; the floor-percentile heuristic is a place where the 4D claim could silently degrade.","The quantization bottleneck that limits depth accuracy suggests that a hybrid discrete-continuous decoder, for example a diffusion head for depth, could close the gap to specialized depth models without sacrificing the cross-modal reasoning.","If these results transfer to in-the-wild footage beyond the benchmarks, downstream augmented-reality and robotics systems could use one model for egomotion, intent, and scene geometry, removing the need to fuse separate specialist outputs."],"forward_implications":["Egocentric body and hand reconstruction no longer needs precomputed camera trajectories, point clouds, or dedicated hand trackers; raw RGB suffices, which widens the range of casual wearable footage the method can process.","Because all modalities share one token sequence, predictions are temporally consistent by construction, so missing or occluded observations, such as hands out of view, can be filled from learned kinematic and scene priors.","Inference at 0.7 seconds per two-second clip puts the reconstruction in a range where real-time or near-real-time wearable assistants are conceivable.","Scaling helps: the ablations show accuracy rises with model size and with the size of the pretraining corpus, and that joint training on data without body annotations still improves body reconstruction.","A metric-aligned 4D coordinate system can be recovered by a simple floor-fitting or anchor step after relative depth prediction, without dense bundle adjustment."],"supporting_citations":[{"why":"Supplies the egocentric multimodal pretraining foundation, the camera and gaze tokenizer architecture, and the masking paradigm that ReViV builds on.","marker":"[22]"},{"why":"Generates the temporally consistent depth pseudo-labels used as geometric supervision for training.","marker":"[5]"},{"why":"Provides the Cosmos tokenizers that quantize RGB and depth videos into discrete tokens for MGET.","marker":"[1]"},{"why":"Serves as the camera-tracking baseline, the metric depth anchor for alignment, and the camera-trajectory provider for body baselines.","marker":"[17]"},{"why":"Provides the unseen Aria Digital Twin benchmark with ground-truth body, camera, gaze, and depth used in the main quantitative comparisons.","marker":"[34]"},{"why":"Diffusion baseline for body reconstruction that ReViV compares against and outperforms without camera input.","marker":"[36]"},{"why":"Diffusion baseline for body and hand estimation that relies on camera trajectories or hand priors; a key comparison target.","marker":"[57]"},{"why":"Frame-based hand reconstruction baseline used in the hand motion comparisons.","marker":"[38]"},{"why":"Optimization-based dynamic hand reconstruction baseline that ReViV outperforms in accuracy and speed.","marker":"[59]"}],"fun_headline_variants":["Single egocentric video reconstructs viewer and scene in 4D","One pass, five modalities: body, hands, gaze, depth, trajectory","ReViV: 4D self and scene from one monocular video, no extras","Fast unified 4D reconstruction from a single first-person clip"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated depth labels and the later floor-fitting or anchor-based scaling are accurate enough to define the metric 4D frame; if these are systematically off, the metric reconstruction claim weakens even if the body and hand pose results stand.","fun_headline_variants_meta":{"raw":{"variants":["Single egocentric video reconstructs viewer and scene in 4D","One pass, five modalities: body, hands, gaze, depth, trajectory","ReViV: 4D self and scene from one monocular video, no extras","Fast unified 4D reconstruction from a single first-person clip"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3122,"prompt_tokens":1024,"completion_tokens":2098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":640,"tokens_out":2098,"duration_ms":12482,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:34:05.159710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take ReViV's predicted depth on the Aria Digital Twin sequences that have ground-truth metric depth, skip the usual clip-wise scale-and-shift alignment, and compare absolute scale; if the scale error is large precisely on sequences where the floor is visible, the floor-fitting step, not the kinematic predictions, is what carries the metric claim.","supporting_citations":[{"cited_title":"In: CVPR (2025)","cited_arxiv_id":null,"evidence_quote":"Generates the temporally consistent depth pseudo-labels used as geometric supervision for training."},{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","cited_arxiv_id":null,"evidence_quote":"Provides the unseen Aria Digital Twin benchmark with ground-truth body, camera, gaze, and depth used in the main quantitative comparisons."},{"cited_title":"In: CVPR (2024)","cited_arxiv_id":null,"evidence_quote":"Frame-based hand reconstruction baseline used in the hand motion comparisons."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2025)","cited_arxiv_id":null,"evidence_quote":"Optimization-based dynamic hand reconstruction baseline that ReViV outperforms in accuracy and speed."}],"review_version":2}